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Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

7 Extensions of lime may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less tlian thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)S Responsive to communication(s) filed on 78 June 2001 . 
2a)\3 This action is FINAL. 2b)l3 This action is non-final. 

3) 0 Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-19 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) ^ Claim(s) 1-19 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification is objected to by the Examiner. 

10) 0 The drawing(s) filed on is/are: a)n accepted or b)n objected to by the Examiner. 

Applicant may not request that any objection to the drawjng{s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction Is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) 0 The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C, § 119 

12) n Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)n All b)n Some * c)^ None of: 

1 .□ Certified copies of the priority documents have been received. 

2.0 Certified copies of the priority documents have been received in Application No. . 

3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 

1 . Claims 1 - 1 9 are presented for examination. 

Claim Rejections - 55 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1-19 are rejected under 35 U.S.C. 103(a) as being unpatentable over Jain et al. 
(Hereinafter "Jain") US Patent No. 5,983,237, in view of Brown US Patent No. 6,61 1,842. 

As to claim 1, Jain teaches the invention substantially as claimed, comprising: 
monitoring one or more multimedia items accessed by a user, each multimedia item 
having two or more disparate modalities [see 132 and 136 of fig. 2], the disparate modalities 
being at least one or more visual modalities and one or more textual modalities [user query and 
image, see fig. 2]; 

creating a visual feature vector for each of the visual modalities and a textual feature 
vector for each of the textual modalities [query vectors, col. 7, lines 32-38]; 

concatenating the visual feature vectors and the textual feature vectors into a unified 
feature vector [feature vectors representative of images, col. 7, lines 39-45]; 
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categorizing each of the multimedia items by categorizing the respective unified feature 
vector [visual categories, col. 65 to col. 16, lines 5; col. 16, lines 59-65; col. 22, lines 24-35]. 

However, Jain does not explicitly teach assembling a user profile based on the 
categorization. Jain teaches assembling a user profile based on the categorization [see the 
abstract; col. 2, lines 59 to col. 3, lines 30]. 

Therefore, it would have been obvious to one of ordinary skill in the Data Processing art 
at the time of the invention to add the feature of Brown to the system of Jain as an essential 
means to classify user personal preferences in term of the categories of products therefore when 
these categories have been identified and the items found falling within such categories can be 
automatically recommended to a user [see Brown, col. 1, lines 35-64]. 

As to claim 2, Jain and Brown teach the invention substantially as claimed. Brown further 
teaches using the user profile to match one or more multimedia items stored in one or more 
databases [col. 2, lines 46-56], 

As to claim 3, Jain and Brown teach the invention substantially as claimed. Brown further 
teaches that where one or more of the databases are part of a computer that is connected to one or 
more networks [col. 2, lines 46-56]. 

As to claim 4, Jain and Brown teach the invention substantially as claimed. Jain further 
teaches that the network includes any one or more of the following: an Internet, an intranet, an 
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extranet, a corporate network, a government network, a infrared network, and a radio frequency 
network [network 100, fig. 1]. 

As to claim 5, Jain and Brown teach the invention substantially as claimed. Brown further 
teaches that the categories include any one or more of the following: a product, a service, an 
interest, a retail item, a hobby, a food item, an item of clothing, a travel package, a vacation 
destination, a financial product, a business partner, a business interest, a medical product, a 
commercial, and a social interest [product, col. 1, lines 16-26]. 

As to claim 6, Jain and Brown teach the invention substantially as claimed. Brown further 
teaches that the category includes any one or more of the following services: consulting, legal, 
real estate, medical, technical, physical training, diet, cosmetic, fashion, governmental, 
automotive, design, architecture, personal assistants, games, on-line games of chance, dating 
services, and landscaping [entertainment media such as the hiring of videos, col. 1, lines 35-45]. 

As to claim 7, Jain and Brown teach the invention substantially as claimed as specified in 
claim 1 above. Jain further teaches concatenating the visual feature vectors and the textual 
feature vectors into a unified feature vector [col. 7, lines 32-46], categorizing each of the 
multimedia items by categorizing the respective unified feature vector [col. 7, lines 39-46]; and 
creating one or more indices of the database based on the categorization [col. 7, Hnes 46-49]. 
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As to claim 8, Jain and Brown teach the invention substantially as claimed. Jain further 
teaches that the database resides on any one or more of the following: a network server, a web 
site, a personal computer, a server farm, and a network disk array [database 108 in the computer 
system 100, fig. 1]. 

As to claim 9, Jain and Brown teach the invention substantially as claimed. Brown further 
teaches making a business decision based on the classifications [see S8-S10 of fig. 4]. 

As to claim 10, Jain and Brown teach the invention substantially as claimed. Brown 
further teaches that the multimedia items are retrievable based on an annotation of the 
classification [data indicative of assessments of the products in a range of different categories, 
col. 1, lines 57-60]. 

As to claim 11, Jain and Brown teach the invention substantially as claimed. Jain further 
teaches that the multimedia items that are retrieved are used as a response to a query of a search 
engine [inherent in the visual information retrieval system of Jain, ab]. 

As to claims 12 and 13, Jain and Brown teach the invention substantially as claimed, with 
the exception of a multimedia e-mail and the multimedia e-mail is routed to any one or more of 
the following: a sender, a folder, a person, a personal folder, a corporate folder, and a corporate 
department. However, this feature is well known in the art such as sending multimedia e-mails to 
the Microsoft Outlook folder. Therefore, it would have been obvious to one of ordinary skill in 
the Data Processing art at the time of the invention to add the feature to Jain-Brown's system as 
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an essential means to expand the diversity maximization method of presenting the search results 
to users. 

As to claim 14, Jain and Brown teach the invention substantially as claimed. Brown 
further teaches that the multimedia e-mail is multiplied before being routed. 

As to claim 15, all limitations of this claim have been addressed in the analysis above, 
and this claim is rejected on that basis. 

As to claim 16, Jain and Brown teach the invention substantially as claimed. Brown 
further teaches that the multimedia items are digital multimedia copies and the feature vectors 
represent restricted multimedia items and the comparison determines if the digital multimedia 
copies are similar to the restricted multimedia items and the decision is that the digital 
multimedia copies are subject to a second restriction [col. 2, lines 46-58]. 

As to claim 1 7, Jain and Brown teach the invention substantially as claimed. Brown 
further teaches that the second restriction includes any one or more of the following: a copyright 
restriction, a trademark restriction, intellectual property restriction, parental guidance restriction, 
conmion decency restrictions, user-defined restrictions [col. 2, lines 59-65]. 
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3. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Thuy Pardo, whose telephone number is (703) 305-1091. The 
examiner can normally be reached Monday through Thursday from 8:30 AM to 5:00 PM. 

If attempts to reach the examiner by telephone are imsuccessful, the examiner's 
supervisor, Dov Popovici, can be reached at (703) 305-3830. 

The fax phone number for the organization where this application or proceeding is 
assigned are as follows: (703) 872-9306 (Official Communication) 

and/or: 

(703) 746-5616 (Use this Fax#, only after approval by Examiner, fi^r 
^^INFORMAL^' or ^^Draft^' communication. Examiner may request that a fifrmal/amendment 
be faxed directly to then on occasions). 

Any inquiry of a general nature of relating to the status of this application should be 
directed to the Group receptionist whose telephone number is (703) 305-3900. 

4. Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
Washington, D.C. 20231 

or faxed to: 

(703) 308-9051, (for formal communications intended for entry) 
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Or: 

(703) 308-5359, (for informal or draft communications, please label 
"PROPOSED" or "DRAFT") 

Hand-delivered responses should be brought to Crystal Park 11, 2121 Crystal 
Drive, Arlington. VA., Sixth Floor (Receptionist). 




Thuy Pardo 
February 05, 2004 



